Unfortunately, this is a known issue. It was first reported here:
NVIDIA/nvidia-container-toolkit#138

The underlying issue is that libnvidia-container injects some devices and modifies some cgroups out-of-band of the container engine it is operating on behalf of when setting a container up for use with GPUs. This causes the internal state of the container engine to be out of sync with what has actually been set up for the container.

For example, if you do a docker inspect on a functioning GPU-enabled container today, you will see that its device list is empty, even though it clearly has the nvidia devices injected into it and the cgroup access to those devices is set up properly.

This has not been an issue until now because everything works fine at initial container creation time. These settings are modified by libnvidia-container only after a container has already been set up by docker and no further updates to the cgroups are necessary.

The problem comes when some external entity hits docker's ContainerUpdate API (whether directly via the CLI or through an API call like the CPUManager in Kubernetes does). When this API is invoked, docker resolves its empty device list to disk, essentially "undoing" what libnvidia-container had set up in regards to these devices.

We need to come up with a solution that allows libnvidia-container to take control of managing these devices on behalf of docker (or any container engine) while properly informing it so that it can keep its internal state in sync.

If your setup is constrained such that GPUs will only ever be used by containers that have CPUSets assigned to them via the static allocation policy, then the following patch to Kubernetes will avoid having docker update its cgroups after these containers are initially launched. This is not a generic enough solution to work in all cases though, unfortunately.

diff --git a/pkg/kubelet/cm/cpumanager/cpu_manager.go b/pkg/kubelet/cm/cpumanager/cpu_manager.go
index 4ccddd5..ff3fbdf 100644
--- a/pkg/kubelet/cm/cpumanager/cpu_manager.go
+++ b/pkg/kubelet/cm/cpumanager/cpu_manager.go
@@ -242,7 +242,8 @@ func (m *manager) reconcileState() (success []reconciledContainer, failure []rec
                        // - policy does not want to track the container
                        // - kubelet has just been restarted - and there is no previous state file
                        // - container has been removed from state by RemoveContainer call (DeletionTimestamp is set)
-                       if _, ok := m.state.GetCPUSet(containerID); !ok {
+                       cset, ok := m.state.GetCPUSet(containerID)
+                       if !ok {
                                if status.Phase == v1.PodRunning && pod.DeletionTimestamp == nil {
                                        klog.V(4).Infof("[cpumanager] reconcileState: container is not present in state - trying to add (pod: %s, container: %s, container id: %s)", pod.Name, container.Name, containerID)
                                        err := m.AddContainer(pod, &container, containerID)
@@ -258,7 +259,13 @@ func (m *manager) reconcileState() (success []reconciledContainer, failure []rec
                                }
                        }

-                       cset := m.state.GetCPUSetOrDefault(containerID)
+                       if !cset.IsEmpty() && m.policy.Name() == string(PolicyStatic) {
+                               klog.V(4).Infof("[cpumanager] reconcileState: skipping container; assigned cpuset unchanged (pod: %s, container: %s, container id: %s, cpuset: \"%v\")", pod.Name, container.Name, containerID, cset)
+                               success = append(success, reconciledContainer{pod.Name, container.Name, containerID})
+                               continue
+                       }
+
+                       cset = m.state.GetDefaultCPUSet()
                        if cset.IsEmpty() {
                                // NOTE: This should not happen outside of tests.
                                klog.Infof("[cpumanager] reconcileState: skipping container; assigned cpuset is empty (pod: %s, container: %s)", pod.Name, container.Name)

@klueska Thanks so much as it works when we use your code snippet and setting CPU POLICY to static and QOS of the container is in "guaranteed" mode.

However, is there anyway to get GPU working for those containers that are not "guaranteed" while CPU_MANAGER_POLICY is set to STATIC. Is it something you guys intend to develop or is it possible for me to work it around.

@klueska As we know, LXD 3.0.0 is already supported for NVIDIA runtime passthrough.
https://discuss.linuxcontainers.org/t/lxd-3-0-0-has-been-released/1491
If there is a case which kubernetes can create LXD3.0.0 containers instead of docker containers, can we bypass the current problem, since LXD3.0.0 is not a ‘docker' and may not require 'libnvidia-container' ?

Any change made to kubernetes is always going to be a workaround. The real fix needs to come in libnvidia-container or docker or some combination of both.

I've never tried LXD with Kubernetes, so I'm not in a position to say how well it would work or not. I do know that LXD still uses libnvidia-container under the hood though, so it may exhibit the same problems.

Again, the underlying problem is that docker is not told about the devices that libnvidia-container injects into it, so if you come up with a workaround that updates docker's internal state with this information, that should be sufficient.

@klueska fixed this in the latest NVIDIA device-plugin (beta5) release. Though this is a workaround.

Note, to use this workaround you will need to use the new daemonset spec nvidia-device-plugin-compat-with-cpumanager.yml instead of the default one.

This spec does two things different from the default one:

1. It adds a new argument to the plugin executable for --pass-device-specs
2. It launches the plugin as --privileged

If you don't want to use the --privileged flag, then things will still "work" in terms of allowing pods with GPUs to run, but you will see the plugin restart anytime a container with guaranteed CPUs from the CPUManager starts. If you are OK with this restart, then launching the daemonset as --privileged is not strictly necessary.

@klueska Is there a PR of device plugin for this issue? I would like to learn about the fix.

https://gitlab.com/nvidia/kubernetes/device-plugin/-/merge_requests/22

@klueska nvidia-device-plugin-compat-with-cpumanager.yml is using nvidia/k8s-device-plugin:v0.7.1. My current cluster is using nvidia/k8s-device-plugin:1.11. Is k8s-device-plugin:v0.7.1 a newer version than k8s-device-plugin:1.11? Can I upgrade the ds directly without breaking things on my production?

@zionwu, Yes v0.7.1 is newer than 1.11 and you should be able to upgrade without any issues.

Please see https://github.com/NVIDIA/k8s-device-plugin#versioning for info on versioning / upgrading.

Also, keep in mind that the semantics around deploying the plugin on nodes that do not have GPUs has slightly changed. You may need to set this flag to false in your daemonset if you rely on the ability to deploy it on non-GPU nodes and not error out:
NVIDIA/k8s-device-plugin@2a9b835

Got it. Thank you ! @klueska